DIMENSION4 trace messages

This section lists the trace messages that might be logged by a DIMENSION4 system, organized as follows:

• The ‘sensor_health’ trace message.  An explanation of the health data periodically sent by sensors.
• Location system information messages enabled by default.  Details of the ‘boot’, ‘ls_sink_stats’, ‘ls_sink_info’, ‘ls_sink_time’, ‘logging_server_stats’ and ‘tftp_report’ messages which are enabled by default in this version of DIMENSION4. 
• Location system information messages disabled by default. Details of message streams that might be useful on occasion but are disabled by default.
• Sensor information messages enabled by default. Details of the ‘sensor_init’, ‘sensor_info’, and ‘sensor_warning’ messages which are enabled by default in this version of DIMENSION4.
• Sensor fatal error messages. A list of all fatal errors that might be sent by sensors.
• Location system low-level debugging messages.  A list of low-level debugging messages sent by the location system, but disabled by default.
• Sensor low-level debugging messages. A list of low-level debugging messages sent by sensors, but disabled by default.
• Location platform warning messages. A list of some of the important warning messages that are logged by the underlying location platform support.

Message format

The format of every message is shown, using the following conventions:

• Constant strings printed in the message are shown as fixed font
• Variables, filled in with some value, are shown as italic (in the health section, they are also underlined).

For example, the format of one of the sensor_init messages is shown as:

And in the log itself this might correspond to a displayed message like this:

Traces enabled by default

From DIMENSION4 version 1.0.2 commonly-required traces are enabled by default, so that no extra user configuration (of the platform_monitor parameter) is required.  If these traces are explicitly enabled, no harm will be done, but the configuration step is no longer required.

The ‘sensor_health’ trace message

ARM section

The ARM section contains health information about the sensor control system, sensor-sensor and sensor-server protocols.  It has several counters that show activity since the previous health report.  These can be used to understand the size of data flows in the system (e.g. bandwidth to server, tag bandwidth at the sensor, sensor to sensor communications).  It also includes more advanced performance data that may be useful if analyzing a subsequent sensor error, but is not valuable for normal field analysis.

This message is sent once a minute.

The health message is broken down as follows:

ARM  

Health data from the sensor CPU (ARM).

|RECV s_arrays/scans/valid/meas/late ScanArrays/Scans/ValidScans/Measurements/Late  

The RECV section gives details about individual measurements the ARM has received over the network or from the digital signal processor (DSP).

ScanArrays: the DSP sends measurements to the ARM in bulk messages called ScanArrays. This value is the number of messages the ARM has received from the DSP.

Scans: the number of measurements the ARM has received from the DSP. For example, if there are four measurements per full ScanArray and the ARM has received six half-full ScanArrays, then the ARM has received twelve measurements.

Valid Scans: the number of measurements which had a valid scan. Approximately half the measurements will not have been scanned because the sensor has a high-resolution decoder which can perform accurate scans to get fine TDOA data and angle data, and a low-resolution decoder which does not. The measurements from the low-resolution decoder are currently not used.

Measurements: the number of measurements received from the network.

Late: the number of measurements (local or received via the network) that were too late to be included in processing for that beacon.

|SENT rem_meas/fltr_sets/fltr_meas Measurements/FilterInputSets/FilterMeasurements  

This SENT section gives details about what the ARM has sent over the network or to the DSP.

Measurements: the number of measurements the ARM has sent to other sensors over the network.

FilterInputSets: the ARM sends measurements to the DSP to be filtered in sets, with each set containing all the measurements gathered for a particular tag beacon event. This value is the number of sets sent to the DSP.

FilterMeasurements: the total number of measurements sent to the DSP to be filtered. For example, if every tag is seen by exactly five sensors, this value should be equal to FilterInputSets multiplied by five.

|FILTER lp_sets/res/seq/drpd_sets SetsDroppedLowPower/FilterResults/FilterSequenceHints/DiscardedFilterSets  

The FILTER section describes the data flows in/out of the filter (from the ARM’s perspective). This is for tags where this sensor is acting as the ‘master’.

SetsDroppedLowPower: the number of sets of messages which were dropped due to power thresholding.

FilterResults: the number of filter results received by the ARM.

FilterSequenceHints: the number of messages which tells the ARM which filter sets have been filtered.

DiscardedFilterSets: the number of filter sets which have been discarded (for example, if the filter queue is too big).

|SENT loc_v2/loc_v3 SentLocationMessagesV2/SentLocationMessagesV3  

This SENT section shows counts of data messages sent to the server.

SentLocationMessagesV2: the number of location messages sent to the location cell manager.

SentLocationMessagesV3: the number of location messages (these include the measurement data) sent to the logging server.

|FAILED meas/loc_v2/loc_v3 SendMeasurementFailures/SendV2Failures/SendV3Failures  

The FAILED section shows counts of low-level network send failures.

SendMeasurementFailures: the number of measurements sent to other sensors which failed to send.

SendV2Failures: the number of location messages sent to the location cell manager which failed to send.

SendV3Failures: the number of location messages sent to the logging server which failed to send.

|STATE alloc_meas/fq_sets/fq_meas AllocatedMeasurements/FilterQueueSets/FilterQueueMeasurements  

The STATE section shows the current state of various buffers.

AllocatedMeasurements: the number of measurements currently allocated.

FilterQueueSets: the number of filter sets which have been sent to the DSP to be filtered for which the ARM is still waiting for a filter result.

FilterQueueMeasurements: the number of measurements in the queued filter sets.

|HWM AllocatedMeasurements/FilterQueueSets/FilterQueueMeasurements  

The HWM section lists high-water-marks (i.e. the highest value seen since the last message) for the values in the STATE message (see above).

|MSGQALLOC: for each channel i with alloc errors: #i: AllocErrors[i]
(if there are no errors) (no errors) 

The MSGQALLOC section gives the total number of errors encountered whilst allocating messages to communicate with the DSP. For each message type, this section gives the number of allocation errors (if a message type has no errors then it is omitted). If there have been no errors then this section has the text “(no errors)”.

MEM HWM/RSS Memory HWM/MemoryResidentSetSize 
or if not able to retrieve memory monitor statistics: (get use err)  

The MEM section gives information on the ARM’s memory usage. If unable to retrieve the usage statistics this section prints an error message.

HWM: the high-water-mark for the memory used by the ARM application.

MemoryResidentSetSize: the current memory used by the ARM application.

| LOOP free/sent/loop_errs/lb_errs SlotsFree/SlotsSent/LoopErrors/LoopBufferErrors

The LOOP section shows the current state of the message loop between the ARM and DSP. The message loop is used to send some messages to/from the DSP. If there are no slots free, then this will cause problems.

SlotsFree: the current number of slots in the message loop which are free (to be allocated).

SlotsSent: the current number of slots in the message loop which have been sent to the DSP.

LoopErrors: the number of errors in the message loop.

LoopBufferErrors: the number of errors in the message loop buffer.

DSP section

The DSP section contains information about the digital signal processor (DSP) and the underlying hardware. In general the data is for advanced analysis only.  However the ‘HEALTH load:’ percentage will give an idea of the remaining processing capacity, and the TS section will give an idea of the performance of the timing subsystem.

This message is sent once a minute.

| DSP 

Health data from the DSP.

|TASKS for each task: TaskNameActivityCount

The TASKS section shows the number of times each task has run. This can be useful to debug unusual situations.

| UPP for each stage in the data pipeline: StageCount  

The UPP section shows the message counts for each stage in the data pipeline from the receiver through to sending the measurement to the ARM.

| FILTER for each stage in the filter pipeline: StageCount

The FILTER section shows the message counts for each stage in the filter pipeline.

| MSGALLOC: for each channel i with alloc errors: #i: AllocErrors[i] 
(if there are no errors) (no errors) 

The MSGALLOC section gives the total number of errors encountered whilst allocating messages to communicate with the DSP. For each message type, this section gives the number of allocation errors (if a message type has no errors then it is omitted). If there have been no errors then this section has the text “(no errors)”.

| HEALTH load: DSPProcessorLoad % 

The HEALTH section shows the current load on the DSP in percent.

LOOP free/sent MessageQueueLoopSlotsFree/MessageQueueLoopSlotsSent

The LOOP section shows the current state of the message loop between the DSP and ARM. The message loop is used to send some messages to/from the ARM. If there are no slots free, then this will cause problems.

MessageQueueLoopSlotsFree: the current number of slots in the message loop which are free (to be allocated).

MessageQueueLoopSlotsSent: the current number of slots in the message loop which have been sent to the ARM.

MEM used/size/blkfree  MemoryUsed/MemorySize/LargestFreeBlock

The MEM section shows the current state of the DSP’s memory.

| DRIFT ClockDriftMicroseconds

The DRIFT section shows the accumulated clock drift in microseconds.

| TS ok/sht/lng NormalTimeslotCount/ShortTimeslotCount/LongTimeslotCount

The TS section shows statistics about the timing subsystem. If ShortTimeslotCount or LongTimeslotCount are increasing, this may indicate a problem.

| CFAR min-max threshold/raw/stripped/real w1-w2/x1-x2/y1-y2/z1-z2

The CFAR section shows the minimum and maximum CFAR values since the last DSP health message for four ranges of values: CFAR threshold w1-w2, raw bits x1-x2, stripped bits y1-y2, and real bits z1-z2.

| RADIO installed
| RADIO absent

If the trace contains the | RADIO installed section then the sensor has a working radio; if the trace has the | RADIO absent section then no radio is installed.

Additional Periodic DSP health information

An additional message is sent out once every three minutes with DSP health information:

|TEMP nrDAC/nrRX TempNearDAC/TempNearReceiver

TempNearDAC: temperature near the DAC in Celsius

TempNearReceiver: temperature near the receiver in Celsius

|DSP|glbl_stack GlobalStackBytesUsed

GlobalStackBytesUsed: number of bytes used in the global stack

|<Task> mode/sp/used/handle TaskMode/TaskStackPtr/TaskUsedStackBytes/TaskHandleAddr

For each task:

TaskMode: the current mode of the task

TaskStackPtr: the stack pointer address

TaskUsedStackBytes: the number of bytes used in the task’s stack

TaskHandleAddr: the address of the task handle

|PPM cnt/no_f/no_l/insuf PacketCount/NoPulsesFirstPart/NoPulsesLastPart/NotEnoughPulses

PacketCount: the count of packets considered

NoPulsesFirstPart: the count of packets with no pulse in the first part of the packet

NoPulsesLastPart: the count of packets with no pulse in the last part of the packet

NotEnoughPulses: the count of packets with not enough pulses

|CABLE unsync/ok/1/>1 UnsyncedBits/ErrorFreeBytes/SingleErrorBytes/MultipleErrorBytes

UnsyncedBits: (unused)

ErrorFreeBytes: bytes which were error free

SingleErrorBytes: bytes which had a single bit error

MultipleErrorBytes: bytes which had multiple bit errors

|LOOP_ERRORS buf/loop TotalLoopBufferErrors/TotalLoopErrors

TotalLoopBufferErrors: the number of errors in the message loop buffer.

TotalLoopErrors: the number of errors in the message loop.

Location system information messages enabled by default

Trace option ‘boot’

This option traces operation of the boot server.  The message formats are:

Trace option ‘ls_sink_info’

This option traces location sink configuration. It can be used to check for whether the persistent or nonpersistent sensor status storage is enabled. The message formats are:

Trace option ‘ls_sink_stats’

This option traces the messages received by the location sink server.  It prints a single trace format that can contain counts for different protocol messages:

Trace option ‘ls_sink_time’

This option traces the time synchronization protocol that is controlled by the location sink.  It has a single message format:

Trace option ‘logging_server_stats’

This option traces the messages received by the logging server (excluding trace messages).  It prints a single trace format that can contain counts for different sensor messages. In the current version the only message type supported is ULocationSystem::Messages::LocationMessageV3-Sensor (i.e. a location message from a sensor).

Trace option ‘tftp_report’

This option gives a report of the TFTP server embedded within the boot server. It has a single message format:

XFERS ok/failed: running total of counts of transfers which successfully completed or failed.

CONNS lwm/curr/hwm: the low-water-mark, current and high-water-mark for the number of active connections. By default the boot server has a maximum of 32 concurrent connections. The lwm and hwm counts are measured since the previous tftp_report message.

RRQ drpd: the number of file transfer requests dropped due to reaching the limit on concurrent connections.

LOSS_INDICATORS rrq_dup/retries/ack_old/ack_fut: various statistics that might indicate lost packets: rrq_dup = duplicate file requests, retries = number of retried server sends, ack_old = number of stale acknowledgement packets, ack_fut = number of extremely old acknowledgement packets.

Location system information messages disabled by default

Trace option ‘ls_upstream_info’

This option traces the timing cable topology discovery process. The message formats are:

Trace option ‘ls_sink_tag_data’

This option traces the receipt of telemetry data from tags.  It has a single message format:

Trace option ‘tftp_info’

This option traces the transfer of boot files using TFTP. It emits about five messages per transfer, and has the following formats:

Sensor information messages enabled by default

Trace option ‘sensor_init’

This option traces sensor initialization.  The message formats are:

Trace option ‘sensor_info’

This option traces various sensor behaviors.  Normal message formats are:

Message formats that probably indicate problems are:

Trace option ‘sensor_warning’

This option traces is used to deliver non-serious warnings about time synchronization.  If network delays are very high, then the network-based time synchronization protocol will lose accuracy.  This is not a problem in this version of the software because a sensor doesn’t use timestamps from other sensors as part of its location protocol, but it is included in case we wish to use timestamps in this way in future versions.  Normal message formats are:

Sensor fatal error messages

These messages are sent by a sensor in response to an unrecoverable error; the sensor will then reboot. Message formats are:

Location system low-level debugging messages

There are many low-level tracing options available whose main purpose is debugging system software operation.  They should not normally be enabled but are included here to give an idea of what could be traced in principle.

Trace option ‘boot_d’

This does extra tracing on the boot protocol.

Trace option ‘ls_sink_liveness_d’

This traces the ‘server liveness timeout’ behavior.

Trace option ‘ls_timing_graph_d’

This traces the calculation of timing graphs.

Trace option ‘ls_timing_delay_checker’

This traces server-side solving of sensor orientations and cable delays.

Trace option ‘ls_referential_integrity’

This traces the object referential integrity checker.

Trace option ‘ls_child_has_timing_issue_checker

This traces computation of the ChildHasTimingIssue flag.

Trace option ‘tftp_report_d’

This option contains all the data in the tftp_report (which is enabled by default) and some additional information for debugging. It has one format:

tftp_report_d: boot_server: for each statistic statistic_name: count/current state

Configuration distribution

There are several distinct trace options for tracing the configuration distribution protocol.

N.B. The config distribution server checks the value of the platform_monitor variable approximately every minute, and updates what trace streams are enabled. This means that the config distribution servers do not need to be restarted to change which trace streams are enabled. Additionally, the config distribution streams (i.e. streams starting with ‘ls_cfgdist’) all output the server’s cell at the start of the message.

Sensor low-level debugging messages

There are several low-level tracing options available whose main purpose is debugging sensor operation.  They should not normally be enabled but are included here to give an idea of what could be traced in principle.

Trace option ‘sensor_cnc’

This is used to trace the custom network configuration protocol.

Trace option ‘sensor_config’

This is used to trace the configuration protocol.

Trace option ‘sensor_sw’

This is used to trace the firmware and software flash update process.

Trace option ‘tftp_sender’

This option traces the TFTP requests that the ARM makes to download new firmware or software to write into flash.

Location platform warning messages

Some important warning messages are provided by the underlying location platform libraries and are common to all Ubisense services, including DIMENSION4 services.

Thread scheduling delays

These events will be reported if a process detects that it is not being scheduled promptly.  For example, if it requests a sleep of a certain length and is in fact woken up some time after the sleep time has expired, then this may cause a warning if the delay in waking the process up is too large.

Disk write latency

Disk write latencies are reported if a process detects that the write operation took too long to complete.